Skip to content

Gate the Ruby version the README counts at, and drop a limitation that is gone - #889

Merged
tannevaled merged 3 commits into
mainfrom
readme-gated
Nov 5, 2026
Merged

tannevaled merged 3 commits into
mainfrom
readme-gated

Conversation

@tannevaled

@tannevaled tannevaled commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

This PR's first commit was built on a false premise and the second undoes
it.
Keeping the record rather than rewriting it, because the correction is
the useful part. The net diff is README.md plus one test file — no workflow
change.

I set out to gate the README's conformance total, believing nothing checked it,
and added a CI job around the published conformancefigure. Then I read
readme_conformance_count_test.go: it has
been checking exactly that for weeks, against BASELINE in the same tree, with
no network and no pinned external version — strictly better than what I was
adding. The job is gone.

What actually was ungated: the version

TestReadmeConformanceCountMatchesTheBaseline checks the total. It does not
check the version that total is counted at, and that version selects the
population counted: the move from 3.4.1 to 4.0.9 changed which
ruby_version_is guards admit examples (+144 passing, 27 gone red). The
same number means a different thing at a different version.

TestReadmeRubyVersionMatchesTheRuntime reads rubyVersion from
internal/vm/kernel_introspect.go and checks it both ways:

what it does what happens when it can't reach a claim
forward every present-tense claim must equal it fewer matches than shapes ⇒ fail
inverse a line naming a different version must declare itself historical exemptions are logged, not silent

Both halves are needed, and a deliberate break proved it. With the table
row set to 4.0.8, the forward check named it — while the inverse check
exempted the same 4.0.8, because that line also says "the move from". A
line-granular marker meant for the version left behind covered a wrong
present-tense claim sitting beside it.

README.md:782 claims the runtime reports 4.0.8, kernel_introspect.go says 4.0.9
README.md:782 names 4.0.8 and is exempt: the line says "the move from"
2 present-tense version claim(s) checked against 4.0.9, 3 historical line(s) exempt

Two README corrections, measured not read

It runs 23,672 → It passes. 23,672 is what passes; rbgo runs
considerably more. The figure was right and the verb was not — which is exactly
why the existing numeric gate was green throughout. No check on a number can
see a wrong verb.

"A backtick command holds the interpreter lock" no longer reproduces. The
probe the README itself carried printed frozen in October and prints ran
now, 5 runs out of 5, and rubydiff reports the program AGREE with MRI
4.0.7 — same output, not merely no-longer-compared output. Removed, with no
cause attributed
: naming one without a control would be a guess dressed as a
fact.

Timeout.timeout was re-checked on the same pass and still fails open —
rbgo prints returned then nil, MRI 4.0.7 prints raised then 0.3 — so it
stays, now with that measurement beside it.

🤖 Generated with Claude Code

tannevaled and others added 2 commits November 4, 2026 15:21
The docs site and the landing page have had conformancefigure for weeks. The
README — the page most people actually read — did not, and it showed.

Pointing the published v0.7.0 tool at this repository reported:

    0 total claim(s) and 0 version claim(s) checked
    conformancefigure: found no statement of the total; either the vocabulary
    no longer matches how the docs phrase it, or the figure was dropped

Zero, on a README that states the total twice. It refused rather than passing,
which is the only reason the gap was visible at all. Three sentences were
phrased so the gate could not reach them, and one of the three was wrong:

* line 16 said **"It runs 23,672"**. 23,672 is what *passes*; rbgo runs
  considerably more — the dated sweep ran 24,116 examples. The figure was
  right and the verb was not, and a hand-maintained README is exactly where
  that survives. Now "It passes", which is also what the docs site says.
* line 1267 said "the total, 23,672, is a derived summary" — an apposition the
  pattern cannot cross. Now "their sum is 23,672,".
* line 782 said "| `RUBY_VERSION` | rbgo reports `"4.0.9"`" — the table pipe
  and "rbgo" sit between the constant and the verb. Now the cell repeats the
  constant.

Nothing was added to the tool's vocabulary for this. The alternative was to
teach it "runs", and that would have taught it to bless a sentence that was
wrong.

After: **2 total claim(s) and 1 version claim(s) checked**, all agreeing.

The new `readme-figure` job passes every input locally
(--baseline/--runtime/--shim). The tool's default is to fetch those three files
from this repository's main branch, which inside this repository is a
self-reference: a pull request that re-records BASELINE and updates the README
in one commit would be refused for disagreeing with the main it is about to
replace.

Verified to bite before committing — README.md:16 set to 23,999:

    README.md:16: claims 23999, the gate records 23672
    conformancefigure: 1 claim(s) disagree with the ruby repository
    exit status 1

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t is gone

This branch started on a false premise, which is worth stating because the
correction is the interesting part. I set out to gate the README's conformance
total, believing nothing checked it, and wired a CI job around the published
`conformancefigure`. Then I read
`TestReadmeConformanceCountMatchesTheBaseline`, which has been checking exactly
that for weeks — against BASELINE in the same tree, with no network and no
pinned external version. My job would have duplicated a better check and added
a dependency. It is not in this commit.

What that test does *not* cover is the version the total is counted at, and
that version selects the population counted: the move from 3.4.1 to 4.0.9
changed which `ruby_version_is` guards admit examples (+144 passing, 27 gone
red). The same number means a different thing at a different version.

TestReadmeRubyVersionMatchesTheRuntime reads `rubyVersion` from
internal/vm/kernel_introspect.go and checks it two ways:

* forward — every present-tense claim ("rbgo reports `"4.0.9"`", "already
  reflect 4.0.9") must equal it, and fewer matches than shapes is a failure, so
  a rewording that escapes the patterns fails instead of silently checking less;
* inverse — a line mentioning RUBY_VERSION and naming a different version must
  declare itself historical ("sweep", "the move from"), and the exemptions are
  logged rather than applied silently.

Both halves are needed, which a deliberate break showed. With the table row set
to 4.0.8 the forward check named it while the inverse check EXEMPTED the same
4.0.8, because that line also says "the move from": a line-granular marker
meant for the version left behind covered a wrong present-tense claim sitting
beside it.

Two README corrections, both measured rather than read:

* line 16 said "It **runs** 23,672". 23,672 is what passes; rbgo runs
  considerably more. The figure was right and the verb was not, which is why
  the existing numeric gate was green the whole time — no check on a number can
  see a wrong verb.
* *"A backtick command holds the interpreter lock"* no longer reproduces. The
  probe the README carried printed `frozen` in October and prints `ran` now,
  five runs out of five, and rubydiff reports the program AGREE with MRI 4.0.7.
  Removed, with no cause attributed: naming one without a control would be a
  guess dressed as a fact.

`Timeout.timeout` was re-checked on the same pass and still fails open — rbgo
prints `returned` then `nil`, MRI 4.0.7 prints `raised` then `0.3` — so it
stays, now with that measurement beside it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled tannevaled changed the title Gate the README's conformance total, which nothing checked Gate the Ruby version the README counts at, and drop a limitation that is gone Nov 4, 2026
@tannevaled

Copy link
Copy Markdown
Contributor Author

Not mergeable, and not because of this change.

test (macos-latest) has been cancelled three times on this commit without
ever acquiring a runner — zero steps, empty log, "runner": "", and attempts 2
and 3 cancelled at 15m01s each, to the second. The other 25 checks are
green. Filed as #890 with the control that rules out the workflow, the label
and the plan: the same job on main at 14:13 UTC got a runner and ran 49
minutes.

Two identical durations are a fixed cap, not a busy pool, so I have stopped
re-running: a fourth attempt would be treating a reproducible cutoff as an
intermittent one. Leaving this open rather than merging on 25 of 26 — the lane
has not said the tests pass, and it has not said they fail either.

@tannevaled
tannevaled merged commit 8055981 into main Nov 5, 2026
26 checks passed
@tannevaled
tannevaled deleted the readme-gated branch November 5, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant